-1- Agent-based Simulations with Historical Data: Issues in Validation

Static estimation treats variation in the dependent data as noise, or error. In simulations using agent-based models, however — especially with dynamic responses — such variation in the simulated output may well possess valuable information from the simulation. This paper will explore previous methods of estimating simulation models (such as indirect inference, the method of simulated moments, and estimation of an auxiliary model), before examining the simulated output from an early agent-based mo del in marketing (Marks Midgley and Cooper 1995), and asking whether these metho ds or others allow the modeller to conclude with some degree of confidence that the simulated output is generated by essentially the same process that generated the historical output by measuring the degree of similarity between two sets of time-series. We intro duce a new measure, the State Similarity Measure (SSM), to measure the distance beween two sets of time-series that embody dynamic responses. Given the degrees of freedom in simulation models, the SSM measure can only increase the confidence in which simulation is used and accepted. 1. Introducing the State Similarity Measure The problem we face is to decide how well an agent-based model we have built in order to examine the rivalrous dance of oligopolistic competition (necessarily with dynamic responses) is performing, compared to historical data of the market we model. This issue is not new. Agent-based models of financial markets — exchange rate markets and markets for financial assets — have large amounts of historical data to compare their mo dels with. Amongst other methods, they have used the “method of simulated moments” in their comparisons. We describe two papers which attempt to use this metho d below. But in order to undertake this method, we need large amounts of data, which are not available for the market we have modelled — brand competition among ground coffee brands in a supermarket chain. This is one issue: given one historical realisation, and given any simulated moments we wish to derive from our model, what is the degree of confidence that the model is capturing the essential actions and reactions of the historical market. Second: just what are the appropriate moments (summary statistics) to calculate for both historical data (when we have enough) and the simulations output? For the exchange